[Tiered Caching] Moving query recomputation logic outside of write lock #14187

sgup432 · 2024-06-11T16:51:11Z

Description

This is related to performance improvement of Tiered Caching. Earlier during a cache miss, we were recomputing the query under the a single write lock which made the other threads wait and degraded performance. This change moves that logic outside the write lock. More details here #13989. We will follow up with another PR to make TieredCache segmented.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
~~[] Public documentation issue/PR created, if applicable.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sagar Upadhyaya <[email protected]>

github-actions · 2024-06-11T17:01:25Z

❌ Gradle check result for b0f6550: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sagar Upadhyaya <[email protected]>

github-actions · 2024-06-11T18:27:48Z

❌ Gradle check result for 1dcb7e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

modules/cache-common/src/main/java/org/opensearch/cache/common/tier/TieredSpilloverCache.java

Signed-off-by: Sagar Upadhyaya <[email protected]>

github-actions · 2024-06-18T07:20:09Z

❌ Gradle check result for 6cf2248: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sgup432 · 2024-06-25T00:41:12Z

Flaky test

[org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndexFailure]

#14292

Signed-off-by: Sagar Upadhyaya <[email protected]>

github-actions · 2024-06-25T02:24:25Z

❕ Gradle check result for 3686d2c: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

sohami · 2024-06-25T19:04:03Z

❕ Gradle check result for 3686d2c: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

#11997

opensearch-trigger-bot · 2024-06-25T19:05:03Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-14187-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 563375de28b16870ab42b9fb4260127598d47d91
# Push it to GitHub
git push --set-upstream origin backport/backport-14187-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-14187-to-2.x.

…ck (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]>

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <[email protected]> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <[email protected]> * Update TESTING.md Co-authored-by: Marc Handalian <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> * Address comment Signed-off-by: bowenlan-amzn <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: Marc Handalian <[email protected]> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * Startreemapping (#24) * Star tree mapping changes with feature flag Signed-off-by: Bharathwaj G <[email protected]> * Fixes and tests Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments and adding validations, integ tests Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments Signed-off-by: Bharathwaj G <[email protected]> --------- Signed-off-by: Bharathwaj G <[email protected]> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed nits Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed major nits Signed-off-by: Sarthak Aggarwal <[email protected]> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: Prudhvi Godithi <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: Bharathwaj G <[email protected]> Signed-off-by: Sarthak Aggarwal <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> Co-authored-by: Prudhvi Godithi <[email protected]> Co-authored-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Co-authored-by: kkewwei <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]>

* Startreemapping (#24) * Star tree mapping changes with feature flag Signed-off-by: Bharathwaj G <[email protected]> * Fixes and tests Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments and adding validations, integ tests Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments Signed-off-by: Bharathwaj G <[email protected]> --------- Signed-off-by: Bharathwaj G <[email protected]> * Onheap impl startree (#25) * Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <[email protected]> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <[email protected]> * Update TESTING.md Co-authored-by: Marc Handalian <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> * Address comment Signed-off-by: bowenlan-amzn <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: Marc Handalian <[email protected]> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * Startreemapping (#24) * Star tree mapping changes with feature flag Signed-off-by: Bharathwaj G <[email protected]> * Fixes and tests Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments and adding validations, integ tests Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments Signed-off-by: Bharathwaj G <[email protected]> --------- Signed-off-by: Bharathwaj G <[email protected]> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed nits Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed major nits Signed-off-by: Sarthak Aggarwal <[email protected]> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: Prudhvi Godithi <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: Bharathwaj G <[email protected]> Signed-off-by: Sarthak Aggarwal <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> Co-authored-by: Prudhvi Godithi <[email protected]> Co-authored-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Co-authored-by: kkewwei <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: Bharathwaj G <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: Prudhvi Godithi <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: Sarthak Aggarwal <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> Co-authored-by: Prudhvi Godithi <[email protected]> Co-authored-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Co-authored-by: kkewwei <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]>

…ck (#14187) (#14565) * Moving query recompute out of write lock * [Tiered Caching] Moving query recomputation logic outside of write lock * Adding java doc for the completable map * Changes to call future handler only once per key * Fixing spotless check * Added changelog * Addressing comments * Fixing gradle fail * Addressing comments to refactor unit test * minor UT refactor --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]>

@internalapi

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <[email protected]> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <[email protected]> * Update TESTING.md Co-authored-by: Marc Handalian <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> * Address comment Signed-off-by: bowenlan-amzn <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: Marc Handalian <[email protected]> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * [AUTO] [main] Add bwc version 2.15.1. (opensearch-project#14549) * Add bwc version 2.15.1 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix auto-generated version Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comment Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Fix file cache initialization (opensearch-project#14004) * fix file cache initialization Signed-off-by: panguixin <[email protected]> * changelog Signed-off-by: panguixin <[email protected]> * add test Signed-off-by: panguixin <[email protected]> --------- Signed-off-by: panguixin <[email protected]> * Add Ashish Singh as maintainer (opensearch-project#14567) Signed-off-by: Bukhtawar Khan <[email protected]> * Allow @internalapi annotation on classes not meant to be constructed outside of the OpenSearch core (opensearch-project#14575) Signed-off-by: Andriy Redko <[email protected]> * Bump com.azure:azure-storage-common from 12.21.2 to 12.25.1 in /plugins/repository-azure (opensearch-project#14517) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.21.2 to 12.25.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-common_12.21.2...azure-storage-blob_12.25.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Add allowlist setting for search-pipeline-common processors (opensearch-project#14562) Add a new static setting that lets an operator choose specific search pipeline processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. A distinct setting exists for each of request, response, and search phase results processors. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Bump Apache Lucene to 9.11.1 (opensearch-project#14576) (opensearch-project#14581) (cherry picked from commit 0095fd1) Signed-off-by: Andriy Redko <[email protected]> * Add unittests for RemoteClusterStateAttributesManager (opensearch-project#14427) * Add unittests for RemoteClusterStateAttributesManager Signed-off-by: Shivansh Arora <[email protected]> * Add Ashish Singh to codeowners (opensearch-project#14592) Signed-off-by: Ashish Singh <[email protected]> * Add batching processor base type AbstractBatchingProcessor (opensearch-project#14554) Signed-off-by: Liyun Xiu <[email protected]> * Add @internalapi annotation to japicmp exclusions (opensearch-project#14597) Signed-off-by: Andriy Redko <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is mis… (opensearch-project#14552) * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix spotless issues. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:update changelog Signed-off-by: vatsal <[email protected]> --------- Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> * Star tree mapping changes (opensearch-project#14605) * Star tree mapping changes with feature flag --------- Signed-off-by: Bharathwaj G <[email protected]> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <[email protected]> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <[email protected]> * rerun security check Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <[email protected]> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Add changelog Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unused logger Signed-off-by: Siddhant Deshmukh <[email protected]> * Spotless apply Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <[email protected]> --------- Signed-off-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <[email protected]> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <[email protected]> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <[email protected]> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <[email protected]> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed nits Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed major nits Signed-off-by: Sarthak Aggarwal <[email protected]> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <[email protected]> * handling for missing doc values Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * rebasing with main Signed-off-by: Sarthak Aggarwal <[email protected]> * support for empty sequential doc values iterator Signed-off-by: Sarthak Aggarwal <[email protected]> * nits Signed-off-by: Sarthak Aggarwal <[email protected]> * min and max star tree aggregators Signed-off-by: Sarthak Aggarwal <[email protected]> * star tree file formats * Star tree codec changes Signed-off-by: Bharathwaj G <[email protected]> * Adding tests Signed-off-by: Bharathwaj G <[email protected]> * Addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments Signed-off-by: Bharathwaj G <[email protected]> * Star tree merge changes Signed-off-by: Bharathwaj G <[email protected]> * fix annotations * star-tree file formats reader and javadoc fixes * read for composite index values Signed-off-by: Sarthak Aggarwal <[email protected]> * doc values file format Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: Prudhvi Godithi <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: panguixin <[email protected]> Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]> Signed-off-by: RS146BIJAY <[email protected]> Signed-off-by: Sooraj Sinha <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Bharathwaj G <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> Co-authored-by: Prudhvi Godithi <[email protected]> Co-authored-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Co-authored-by: kkewwei <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Co-authored-by: panguixin <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Shivansh Arora <[email protected]> Co-authored-by: Ashish Singh <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: Vatsal <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> Co-authored-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> Co-authored-by: rishavz_sagar <[email protected]> Co-authored-by: Sooraj Sinha <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]>

@internalapi

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <[email protected]> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <[email protected]> * Update TESTING.md Co-authored-by: Marc Handalian <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> * Address comment Signed-off-by: bowenlan-amzn <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: Marc Handalian <[email protected]> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * [AUTO] [main] Add bwc version 2.15.1. (opensearch-project#14549) * Add bwc version 2.15.1 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix auto-generated version Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comment Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Fix file cache initialization (opensearch-project#14004) * fix file cache initialization Signed-off-by: panguixin <[email protected]> * changelog Signed-off-by: panguixin <[email protected]> * add test Signed-off-by: panguixin <[email protected]> --------- Signed-off-by: panguixin <[email protected]> * Add Ashish Singh as maintainer (opensearch-project#14567) Signed-off-by: Bukhtawar Khan <[email protected]> * Allow @internalapi annotation on classes not meant to be constructed outside of the OpenSearch core (opensearch-project#14575) Signed-off-by: Andriy Redko <[email protected]> * Bump com.azure:azure-storage-common from 12.21.2 to 12.25.1 in /plugins/repository-azure (opensearch-project#14517) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.21.2 to 12.25.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-common_12.21.2...azure-storage-blob_12.25.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Add allowlist setting for search-pipeline-common processors (opensearch-project#14562) Add a new static setting that lets an operator choose specific search pipeline processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. A distinct setting exists for each of request, response, and search phase results processors. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Bump Apache Lucene to 9.11.1 (opensearch-project#14576) (opensearch-project#14581) (cherry picked from commit 0095fd1) Signed-off-by: Andriy Redko <[email protected]> * Add unittests for RemoteClusterStateAttributesManager (opensearch-project#14427) * Add unittests for RemoteClusterStateAttributesManager Signed-off-by: Shivansh Arora <[email protected]> * Add Ashish Singh to codeowners (opensearch-project#14592) Signed-off-by: Ashish Singh <[email protected]> * Add batching processor base type AbstractBatchingProcessor (opensearch-project#14554) Signed-off-by: Liyun Xiu <[email protected]> * Add @internalapi annotation to japicmp exclusions (opensearch-project#14597) Signed-off-by: Andriy Redko <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is mis… (opensearch-project#14552) * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix spotless issues. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:update changelog Signed-off-by: vatsal <[email protected]> --------- Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> * Star tree mapping changes (opensearch-project#14605) * Star tree mapping changes with feature flag --------- Signed-off-by: Bharathwaj G <[email protected]> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <[email protected]> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <[email protected]> * rerun security check Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <[email protected]> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Add changelog Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unused logger Signed-off-by: Siddhant Deshmukh <[email protected]> * Spotless apply Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <[email protected]> --------- Signed-off-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <[email protected]> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <[email protected]> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <[email protected]> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <[email protected]> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed nits Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed major nits Signed-off-by: Sarthak Aggarwal <[email protected]> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <[email protected]> * handling for missing doc values Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * rebasing with main Signed-off-by: Sarthak Aggarwal <[email protected]> * support for empty sequential doc values iterator Signed-off-by: Sarthak Aggarwal <[email protected]> * nits Signed-off-by: Sarthak Aggarwal <[email protected]> * min and max star tree aggregators Signed-off-by: Sarthak Aggarwal <[email protected]> * star tree file formats * Star tree codec changes Signed-off-by: Bharathwaj G <[email protected]> * Adding tests Signed-off-by: Bharathwaj G <[email protected]> * Addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments Signed-off-by: Bharathwaj G <[email protected]> * Star tree merge changes Signed-off-by: Bharathwaj G <[email protected]> * fix annotations * star-tree file formats reader and javadoc fixes * read for composite index values Signed-off-by: Sarthak Aggarwal <[email protected]> * doc values file format Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: Prudhvi Godithi <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: panguixin <[email protected]> Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]> Signed-off-by: RS146BIJAY <[email protected]> Signed-off-by: Sooraj Sinha <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Bharathwaj G <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> Co-authored-by: Prudhvi Godithi <[email protected]> Co-authored-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Co-authored-by: kkewwei <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Co-authored-by: panguixin <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Shivansh Arora <[email protected]> Co-authored-by: Ashish Singh <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: Vatsal <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> Co-authored-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> Co-authored-by: rishavz_sagar <[email protected]> Co-authored-by: Sooraj Sinha <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]>

@internalapi

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <[email protected]> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <[email protected]> * Update TESTING.md Co-authored-by: Marc Handalian <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> * Address comment Signed-off-by: bowenlan-amzn <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: Marc Handalian <[email protected]> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * [AUTO] [main] Add bwc version 2.15.1. (opensearch-project#14549) * Add bwc version 2.15.1 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix auto-generated version Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comment Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Fix file cache initialization (opensearch-project#14004) * fix file cache initialization Signed-off-by: panguixin <[email protected]> * changelog Signed-off-by: panguixin <[email protected]> * add test Signed-off-by: panguixin <[email protected]> --------- Signed-off-by: panguixin <[email protected]> * Add Ashish Singh as maintainer (opensearch-project#14567) Signed-off-by: Bukhtawar Khan <[email protected]> * Allow @internalapi annotation on classes not meant to be constructed outside of the OpenSearch core (opensearch-project#14575) Signed-off-by: Andriy Redko <[email protected]> * Bump com.azure:azure-storage-common from 12.21.2 to 12.25.1 in /plugins/repository-azure (opensearch-project#14517) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.21.2 to 12.25.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-common_12.21.2...azure-storage-blob_12.25.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Add allowlist setting for search-pipeline-common processors (opensearch-project#14562) Add a new static setting that lets an operator choose specific search pipeline processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. A distinct setting exists for each of request, response, and search phase results processors. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Bump Apache Lucene to 9.11.1 (opensearch-project#14576) (opensearch-project#14581) (cherry picked from commit 0095fd1) Signed-off-by: Andriy Redko <[email protected]> * Add unittests for RemoteClusterStateAttributesManager (opensearch-project#14427) * Add unittests for RemoteClusterStateAttributesManager Signed-off-by: Shivansh Arora <[email protected]> * Add Ashish Singh to codeowners (opensearch-project#14592) Signed-off-by: Ashish Singh <[email protected]> * Add batching processor base type AbstractBatchingProcessor (opensearch-project#14554) Signed-off-by: Liyun Xiu <[email protected]> * Add @internalapi annotation to japicmp exclusions (opensearch-project#14597) Signed-off-by: Andriy Redko <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is mis… (opensearch-project#14552) * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix spotless issues. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:update changelog Signed-off-by: vatsal <[email protected]> --------- Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> * Star tree mapping changes (opensearch-project#14605) * Star tree mapping changes with feature flag --------- Signed-off-by: Bharathwaj G <[email protected]> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <[email protected]> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <[email protected]> * rerun security check Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <[email protected]> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Add changelog Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unused logger Signed-off-by: Siddhant Deshmukh <[email protected]> * Spotless apply Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <[email protected]> --------- Signed-off-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <[email protected]> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <[email protected]> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <[email protected]> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <[email protected]> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed nits Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed major nits Signed-off-by: Sarthak Aggarwal <[email protected]> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <[email protected]> * handling for missing doc values Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * rebasing with main Signed-off-by: Sarthak Aggarwal <[email protected]> * support for empty sequential doc values iterator Signed-off-by: Sarthak Aggarwal <[email protected]> * nits Signed-off-by: Sarthak Aggarwal <[email protected]> * min and max star tree aggregators Signed-off-by: Sarthak Aggarwal <[email protected]> * star tree file formats * Star tree codec changes Signed-off-by: Bharathwaj G <[email protected]> * Adding tests Signed-off-by: Bharathwaj G <[email protected]> * Addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments Signed-off-by: Bharathwaj G <[email protected]> * Star tree merge changes Signed-off-by: Bharathwaj G <[email protected]> * fix annotations * star-tree file formats reader and javadoc fixes * read for composite index values Signed-off-by: Sarthak Aggarwal <[email protected]> * doc values file format Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: Prudhvi Godithi <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: panguixin <[email protected]> Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]> Signed-off-by: RS146BIJAY <[email protected]> Signed-off-by: Sooraj Sinha <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Bharathwaj G <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> Co-authored-by: Prudhvi Godithi <[email protected]> Co-authored-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Co-authored-by: kkewwei <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Co-authored-by: panguixin <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Shivansh Arora <[email protected]> Co-authored-by: Ashish Singh <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: Vatsal <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> Co-authored-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> Co-authored-by: rishavz_sagar <[email protected]> Co-authored-by: Sooraj Sinha <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]>

…ck (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]>

…ck (opensearch-project#14187) (opensearch-project#14565) * Moving query recompute out of write lock * [Tiered Caching] Moving query recomputation logic outside of write lock * Adding java doc for the completable map * Changes to call future handler only once per key * Fixing spotless check * Added changelog * Addressing comments * Fixing gradle fail * Addressing comments to refactor unit test * minor UT refactor --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Signed-off-by: kkewwei <[email protected]>

…ck (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]>

commit 93a3132 Author: Peter Alfonsi <[email protected]> Date: Wed Sep 4 11:44:03 2024 -0700 fix broken ITs Signed-off-by: Peter Alfonsi <[email protected]> commit fd92c5a Author: Marc Handalian <[email protected]> Date: Thu Jan 11 06:41:55 2024 -0800 Update runTask to optionally install plugins (opensearch-project#11844) Signed-off-by: Marc Handalian <[email protected]> commit 4d78661 Author: Peter Alfonsi <[email protected]> Date: Tue Sep 3 16:02:31 2024 -0700 Misc test fixes Signed-off-by: Peter Alfonsi <[email protected]> commit 70a6935 Author: Peter Alfonsi <[email protected]> Date: Fri Aug 23 10:42:05 2024 -0700 [Bugfix] Fixes IRC NPE bug for timed-out cacheable queries (opensearch-project#15327) * Fix IRC timeout bug Signed-off-by: Peter Alfonsi <[email protected]> * addressed Sagar's comments Signed-off-by: Peter Alfonsi <[email protected]> * addressed Ankit's comments Signed-off-by: Peter Alfonsi <[email protected]> * Add UT for test coverage Signed-off-by: Peter Alfonsi <[email protected]> * rerun gradle Signed-off-by: Peter Alfonsi <[email protected]> * tweak imports in new UT Signed-off-by: Peter Alfonsi <[email protected]> * rerun gradle Signed-off-by: Peter Alfonsi <[email protected]> * rerun gradle Signed-off-by: Peter Alfonsi <[email protected]> * rerun gradle Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit f9943f8 Author: Sagar <[email protected]> Date: Wed Jul 17 19:02:05 2024 -0700 Clear ehcache disk cache files during initialization (opensearch-project#14738) * Clear ehcache disk cache files during initialization Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding UT to fix line coverage Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comment Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding more Uts for better line coverage Signed-off-by: Sagar Upadhyaya <[email protected]> * Throwing exception in case we fail to clear cache files during startup Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding more UTs Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding a UT for more coverage Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle build Signed-off-by: Sagar Upadhyaya <[email protected]> * Update ehcache disk cache close() logic Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> commit fc2007e Author: Peter Alfonsi <[email protected]> Date: Mon Jul 1 15:54:39 2024 -0700 [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <[email protected]> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <[email protected]> * rerun security check Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit 1ef62e2 Author: Sagar <[email protected]> Date: Wed Jun 26 11:14:42 2024 -0700 Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comment Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> commit 29ec9d6 Author: Sagar <[email protected]> Date: Tue Jun 25 12:04:17 2024 -0700 [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> commit 8347779 Author: Peter Alfonsi <[email protected]> Date: Tue Jun 25 13:26:54 2024 -0700 Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit 40f868d Author: Kiran Prakash <[email protected]> Date: Thu Jun 20 14:17:23 2024 -0700 Update IndicesRequestCacheCleanupIT.java (opensearch-project#14478) Signed-off-by: Kiran Prakash <[email protected]> commit 094a9f2 Author: Kiran Prakash <[email protected]> Date: Wed Jun 19 15:36:59 2024 -0700 Fix Indices Request Cache ITs from flaking and timing out (opensearch-project#14369) Signed-off-by: Kiran Prakash <[email protected]> commit 3a110a8 Author: Kiran Prakash <[email protected]> Date: Tue Jun 18 10:18:25 2024 -0700 [Tiered Cache] Use ConcurrentHashMap explicitly in IndicesRequestCache (opensearch-project#14409) Signed-off-by: Kiran Prakash <[email protected]> commit bcf0f59 Author: Peter Alfonsi <[email protected]> Date: Fri Jun 14 13:20:51 2024 -0700 [Bugfix] Fix TieredSpilloverCache flaky tests (opensearch-project#14333) * Fix flaky TSC stats tests Signed-off-by: Peter Alfonsi <[email protected]> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <[email protected]> * fix forbidden API Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit a222a03 Author: Kiran Prakash <[email protected]> Date: Wed Jun 12 18:30:06 2024 -0700 [Tiered Caching] [Bug Fix] Use concurrentMap instead of HashMap to fix Concurrent Modification Exception (opensearch-project#14221) * use concurrentmap Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * revert feature flags Signed-off-by: Kiran Prakash <[email protected]> * changelog to releaselog Signed-off-by: Kiran Prakash <[email protected]> * use concurrentmap Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * revert feature flags Signed-off-by: Kiran Prakash <[email protected]> * changelog to releaselog Signed-off-by: Kiran Prakash <[email protected]> * revert the test removal Signed-off-by: Kiran Prakash <[email protected]> * revert the conflict resolutions Signed-off-by: Kiran Prakash <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]> commit f85cd00 Author: Sagar <[email protected]> Date: Wed Jun 12 08:22:38 2024 -0700 Fix ShardNotFoundException during request cache clean up (opensearch-project#14219) * Fix for ShardNotFoundException during request cache clean up Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Fix forbidden gradle check Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> commit 406ae78 Author: Sagar <[email protected]> Date: Thu Jun 6 15:02:00 2024 -0700 [Caching] Move cache removal notifications outside lru lock (opensearch-project#14017) --------- Signed-off-by: Sagar Upadhyaya <[email protected]> commit c5fd7fe Author: Peter Alfonsi <[email protected]> Date: Tue Sep 3 14:38:26 2024 -0700 Change IRC UT to correctly create shard Signed-off-by: Peter Alfonsi <[email protected]> commit 9c4d4af Author: Peter Alfonsi <[email protected]> Date: Tue Jun 4 16:09:18 2024 -0700 [Tiered Caching] Additional ITs for cache stats (opensearch-project#13655) * Adds cache clear IT Signed-off-by: Peter Alfonsi <[email protected]> Cleaned up logic for cache stats ITs Signed-off-by: Peter Alfonsi <[email protected]> Adds more tests around tiered spillover cache Signed-off-by: Peter Alfonsi <[email protected]> Fixed cache stats behavior for overall /_nodes/stats call Signed-off-by: Peter Alfonsi <[email protected]> cleanup Signed-off-by: Peter Alfonsi <[email protected]> Fixed folder structure Signed-off-by: Peter Alfonsi <[email protected]> Addressed Sagar's comments Signed-off-by: Peter Alfonsi <[email protected]> Addressed Ankit's comments Signed-off-by: Peter Alfonsi <[email protected]> Break horrifyingly long test case into many shorter cases Signed-off-by: Peter Alfonsi <[email protected]> Added unsupported operation exception to TSC stats holder incrementEvictions() Signed-off-by: Peter Alfonsi <[email protected]> Addressed Sorabh's comments Signed-off-by: Peter Alfonsi <[email protected]> * rerun assemble Signed-off-by: Peter Alfonsi <[email protected]> * rerun gradle Signed-off-by: Peter Alfonsi <[email protected]> * rerun gradle Signed-off-by: Peter Alfonsi <[email protected]> * rerun gradle Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit be97e90 Author: Peter Alfonsi <[email protected]> Date: Thu May 23 12:02:22 2024 -0700 [Bugfix] [Tiered Caching] Fixes issues when integrating tiered cache with disk cache (opensearch-project#13784) --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit 60ee715 Author: Kiran Prakash <[email protected]> Date: Mon May 20 12:52:03 2024 -0700 Update IndicesRequestCacheIT.java (opensearch-project#13678) Signed-off-by: Kiran Prakash <[email protected]> commit 51f6c1b Author: Sagar <[email protected]> Date: Wed May 8 09:15:36 2024 -0700 Fix IndicesRequestCache clean up logic (opensearch-project#13597) Signed-off-by: Sagar Upadhyaya <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> commit 49e2701 Author: Sagar <[email protected]> Date: Mon May 6 17:25:45 2024 -0700 Fix negative requestStats memory_size issue (opensearch-project#13553) This solves the bug where RequestStats memory_size metric was going negative in certain scenarios as reported in the issue. It turns out that the issue occurs when an indexShard is deleted and then reallocated on the same node. So whenever stale entries from older shard are deleted, those are accounted for the new shard which has the same shardId. --------- Signed-off-by: Sagar Upadhyaya <[email protected]> commit a5c5675 Author: Peter Alfonsi <[email protected]> Date: Thu May 2 17:29:07 2024 -0700 [Tiered Caching] Adds stats implementation for TieredSpilloverCache (opensearch-project#13236) Stats rework part 4 of 4 --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit 46944cb Author: peteralfonsi <[email protected]> Date: Tue Apr 30 10:59:03 2024 -0700 Fix flaky test CacheStatsAPIIndicesRequestCacheIT.testNullLevels() (opensearch-project#13457) * Fix flaky test Signed-off-by: Peter Alfonsi <[email protected]> * Initialize CommonStatsFlags with empty array for levels Signed-off-by: Peter Alfonsi <[email protected]> * Fixes tests using incorrect null levels Signed-off-by: Peter Alfonsi <[email protected]> * rerun Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit 71fcad4 Author: Kiran Prakash <[email protected]> Date: Mon Apr 29 18:41:33 2024 -0700 Fix Flaky test IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices (opensearch-project#13453) * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]> commit f60339f Author: peteralfonsi <[email protected]> Date: Mon Apr 29 22:37:05 2024 -0700 [Tiered Caching] Bump versions for serialization in new cache stats API (opensearch-project#13460) --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit 88718a2 Author: peteralfonsi <[email protected]> Date: Mon Apr 29 14:47:52 2024 -0700 [Tiered Caching] Expose new cache stats API (opensearch-project#13237) Step 3 out of 4 --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit 5e97082 Author: peteralfonsi <[email protected]> Date: Sun Apr 28 08:15:18 2024 -0700 [Tiered Caching] Gate CacheStatsHolder logic behind FeatureFlags.PLUGGABLE_CACHE setting (opensearch-project#13238) Stats rework step 2 of 4 --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit ed881f7 Author: Kiran Prakash <[email protected]> Date: Fri Apr 26 11:52:31 2024 -0700 [Tiered Caching] Make Indices Request Cache Stale Key Mgmt Threshold setting dynamic (opensearch-project#12941) * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update ClusterSettings.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * spotless Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * some refactoring Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * address existing tests Signed-off-by: Kiran Prakash <[email protected]> * UTs Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * ITs Signed-off-by: Kiran Prakash <[email protected]> * spotless Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * resolve conflicts Signed-off-by: Kiran Prakash <[email protected]> * address code comments Signed-off-by: Kiran Prakash <[email protected]> * address code comments Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * rename tests Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> * resolve conflicts Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * code comments Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheIT.java Signed-off-by: Kiran Prakash <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]> commit 424ccc9 Author: Sagar <[email protected]> Date: Thu Apr 25 22:09:42 2024 -0700 [Tiered Caching] Expose a dynamic setting to disable/enable disk cache (opensearch-project#13373) * [Tiered Caching] Expose a dynamic setting to disable/enable disk cache Signed-off-by: Sagar Upadhyaya <[email protected]> * Putting tiered cache settings behind feature flag Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding a changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing Sorabh's comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Putting new setting behind feature flag Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> commit acbbb2f Author: Kiran Prakash <[email protected]> Date: Wed Apr 24 14:50:39 2024 -0700 [Tiered Caching] Bug fix for IndicesRequestCache StaleKey management (opensearch-project#13070) * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * revert Signed-off-by: Kiran Prakash <[email protected]> * revert Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * code comments only Signed-off-by: Kiran Prakash <[email protected]> * docs changes Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * revert catching AlreadyClosedException Signed-off-by: Kiran Prakash <[email protected]> * assert Signed-off-by: Kiran Prakash <[email protected]> * conflicts Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCacheTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * address comments Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * Update IndicesRequestCache.java Signed-off-by: Kiran Prakash <[email protected]> * address conflicts Signed-off-by: Kiran Prakash <[email protected]> * spotless apply Signed-off-by: Kiran Prakash <[email protected]> * address comments Signed-off-by: Kiran Prakash <[email protected]> * update code comments Signed-off-by: Kiran Prakash <[email protected]> * address bug & add tests Signed-off-by: Kiran Prakash <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]> commit f1d2e72 Author: Sagar <[email protected]> Date: Mon Apr 15 10:43:22 2024 -0800 [Tiered Caching] Ehcache Disk cache IT (opensearch-project#12904) * Ehcache IT tests Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding some logs to print key/value size Signed-off-by: Sagar Upadhyaya <[email protected]> * Add ehcache related invalidation IT Signed-off-by: Sagar Upadhyaya <[email protected]> * Remvoing unnecessary IndicesRequestCache IT Signed-off-by: Sagar Upadhyaya <[email protected]> * Indentation fix Signed-off-by: Sagar Upadhyaya <[email protected]> * Added tests around expiration time and invalidation Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> commit b0de406 Author: peteralfonsi <[email protected]> Date: Fri Apr 12 16:09:06 2024 -0700 [Tiered Caching] Stats rework (1/3): Interfaces and implementations for individual tiers (opensearch-project#12531) As part of tiered caching stats, changes the common ICache interface to use ICacheKey as its key. This key contains dimensions (for example, shard ID, index name, or tier) that can be used to aggregate stats. Also changes the CacheStats interface to store the necessary cache stats, and to support getting stats either as a total or aggregated by these dimensions. Integrates these changes with OpenSearchOnHeapCache and EhcacheDiskCache. The stats implementation for the TieredSpilloverCache will be in a followup PR. --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit 40a8d34 Author: Sagar <[email protected]> Date: Thu Apr 11 14:49:03 2024 -0800 [Tiered Caching] Make took time policy dynamic and add additional integ tests (opensearch-project#13063) --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> commit 6f88777 Author: Sagar <[email protected]> Date: Tue Mar 19 19:41:59 2024 -0700 Fixing ehcache flaky test (opensearch-project#12764) * Fixing ehcache flaky test Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding a ehcache issue reference for thread leak issue Signed-off-by: Sagar Upadhyaya <[email protected]> * Updating comment Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> commit f69c089 Author: Sagar <[email protected]> Date: Mon Mar 18 22:20:52 2024 -0700 [Tiered Caching] Clear up disk cache(ehcache) files during node shutdown (opensearch-project#12734) * Adding logic to clear up the disk cache files during close() * Adding logic to update entries count after invalidateAll() * Removing unneeded system log statement * Added comment in test for readability * Fixing issue where we were sending compacted byte[] array to ehcache but calculating size with padded byte[] --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> commit bf6488d Author: Sagar <[email protected]> Date: Mon Mar 18 22:17:14 2024 -0700 [Tiered Caching] Fix test testComputeIfAbsentWithFactoryBasedCacheCreation (opensearch-project#12700) --------- Signed-off-by: Sagar Upadhyaya <[email protected]> commit 0fc1f40 Author: peteralfonsi <[email protected]> Date: Mon Mar 18 13:39:35 2024 -0700 [Tiered Caching] Serializers for ehcache (opensearch-project#12709) Adds serializers and integrates them into ehcache disk cache --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit a45be0f Author: Kiran Prakash <[email protected]> Date: Sat Mar 16 17:12:27 2024 -0700 [Tiered Caching] Indices Request cache stalekey management (opensearch-project#12625) * Introduce IndicesRequestCacheCleanupManager Signed-off-by: Kiran Prakash <[email protected]> * using cleanup mgr to enqueue cleanups Signed-off-by: Kiran Prakash <[email protected]> * readability improvements Signed-off-by: Kiran Prakash <[email protected]> * update updateCleanupKeyToCountMap on new cache entry Signed-off-by: Kiran Prakash <[email protected]> * create IndicesRequestCacheCleanupManager & settings and validators Signed-off-by: Kiran Prakash <[email protected]> * Add IRC specific cache cleaner and remove from IndicesService Signed-off-by: Kiran Prakash <[email protected]> * Unit Tests Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * move cachecleaner inside mgr Signed-off-by: Kiran Prakash <[email protected]> * remove processCleanupKeys Signed-off-by: Kiran Prakash <[email protected]> * minor cleanups Signed-off-by: Kiran Prakash <[email protected]> * add updateCleanupKeyToCountMapOnCacheEviction Signed-off-by: Kiran Prakash <[email protected]> * remove locks and make all methods synchronized Signed-off-by: Kiran Prakash <[email protected]> * spotless Signed-off-by: Kiran Prakash <[email protected]> * updateCleanupKeyToCountMapOnCacheEviction Signed-off-by: Kiran Prakash <[email protected]> * Testing Signed-off-by: Kiran Prakash <[email protected]> * add Reschedule back to indices service Signed-off-by: Kiran Prakash <[email protected]> * rename updateStaleKeysCount to incrementStaleKeysCount Signed-off-by: Kiran Prakash <[email protected]> * rename getStaleKeysCountForTesting to getStaleKeysCount Signed-off-by: Kiran Prakash <[email protected]> * rename threshold to stalenessThreshold Signed-off-by: Kiran Prakash <[email protected]> * check for cleanupKey.entity == null Signed-off-by: Kiran Prakash <[email protected]> * use computeIfPresent with keycountmap Signed-off-by: Kiran Prakash <[email protected]> * log both staleKeysInCache & Staleness in debug logs Signed-off-by: Kiran Prakash <[email protected]> * Use HashMap instead of ConcurrentMap Signed-off-by: Kiran Prakash <[email protected]> * Address b/w compatibility Signed-off-by: Kiran Prakash <[email protected]> * remove synchronized for updateCleanupKeyToCountMapOnCacheEviction Signed-off-by: Kiran Prakash <[email protected]> * make cleanCache synchronized Signed-off-by: Kiran Prakash <[email protected]> * remove shouldRemoveKey Signed-off-by: Kiran Prakash <[email protected]> * spotlessApply Signed-off-by: Kiran Prakash <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]> commit bad412c Author: Sagar <[email protected]> Date: Fri Mar 15 16:53:54 2024 -0700 [Tiered caching] Supporting removal function on EhcacheDiskCache iterator (opensearch-project#12653) * [Tiered caching] Supporting removal function on EhcacheDiskCache iterator Signed-off-by: Sagar Upadhyaya <[email protected]> * Minor refactoring in unit test Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> commit 15c4afa Author: peteralfonsi <[email protected]> Date: Fri Mar 15 15:36:22 2024 -0700 [Tiered Caching] Cache tier policies (opensearch-project#12542) * Adds policy interface and took time policy impl Signed-off-by: Peter Alfonsi <[email protected]> * Changes IndicesService to write a CachePolicyInfoWrapper before the QSR Signed-off-by: Peter Alfonsi <[email protected]> * Moved took time logic from QSR to IndicesService Signed-off-by: Peter Alfonsi <[email protected]> * spotlessApply Signed-off-by: Peter Alfonsi <[email protected]> * Addressed ansjcy's comments Signed-off-by: Peter Alfonsi <[email protected]> * Partial rebase on most recent changes Signed-off-by: Peter Alfonsi <[email protected]> * Integrated policies with new TSC changes Signed-off-by: Peter Alfonsi <[email protected]> * Reverted unintended change to idea/vcs.xml Signed-off-by: Peter Alfonsi <[email protected]> * javadocs Signed-off-by: Peter Alfonsi <[email protected]> * github actions Signed-off-by: Peter Alfonsi <[email protected]> * Set default threshold value to 10 ms Signed-off-by: Peter Alfonsi <[email protected]> * Addressed Sorabh's comments Signed-off-by: Peter Alfonsi <[email protected]> * Addressed Sorabh's second round of comments Signed-off-by: Peter Alfonsi <[email protected]> * Set cachedQueryParser in IRC Signed-off-by: Peter Alfonsi <[email protected]> * Addressed Sorabh's comments besides dynamic setting Signed-off-by: Peter Alfonsi <[email protected]> * Removed dynamic setting, misc comments Signed-off-by: Peter Alfonsi <[email protected]> * Added changelog entry Signed-off-by: Peter Alfonsi <[email protected]> * Added missing javadoc Signed-off-by: Peter Alfonsi <[email protected]> * Fixed failed gradle run Signed-off-by: Peter Alfonsi <[email protected]> * Added setting validation test Signed-off-by: Peter Alfonsi <[email protected]> * rerun gradle for flaky IT Signed-off-by: Peter Alfonsi <[email protected]> * javadocs Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> commit 45fce15 Author: Sagar <[email protected]> Date: Thu Mar 14 13:00:24 2024 -0700 [Tiered Caching] Fixing flaky tiered cache test (opensearch-project#12650) * Fixing flaky tiered cache test Signed-off-by: Sagar Upadhyaya <[email protected]> * Removing unnecessary comment Signed-off-by: Sagar Upadhyaya <[email protected]> * Removing unused variable Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> commit bb66699 Author: Peter Alfonsi <[email protected]> Date: Fri Aug 30 11:36:05 2024 -0700 Attempt to fix broken ITs Signed-off-by: Peter Alfonsi <[email protected]> commit 58170bf Author: Sagar <[email protected]> Date: Mon Mar 11 14:12:55 2024 -0700 [Tiered caching] Integrating IndicesRequestCache with CacheService controlled by a feature flag (opensearch-project#12533) * Adding changelog * Fixing gradle build issue * Fixing CacheService test * Adding UT in IndicesRequestCache with feature flag for more coverage * Updating changelog and renaming feature flag setting * Moving feature flag setting handling logic to CacheService by maintaining backward compatibility * Fixing broken UTs --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> commit 80d2130 Author: Sagar <[email protected]> Date: Fri Mar 1 11:10:28 2024 -0800 [Tiered caching] Introducing cache plugins and exposing Ehcache as one of the pluggable disk cache option (opensearch-project#11874) Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> commit f372c2c Author: Sagar <[email protected]> Date: Wed Jan 10 17:06:29 2024 -0800 [Tiered Caching] Enable serialization of IndicesRequestCache.Key (opensearch-project#10275) --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Kiran Prakash <[email protected]> commit b880c0b Author: Sagar <[email protected]> Date: Tue Jan 9 10:57:52 2024 -0800 [Tiered caching] Framework changes (opensearch-project#10753) * [Tiered caching] Framework changes Signed-off-by: Sagar Upadhyaya <[email protected]> * Added javadoc for new files/packages Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing javadoc warnings Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing additional minor comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Moving non null check to builder for OS onHeapCache Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding package-info for new packages Signed-off-by: Sagar Upadhyaya <[email protected]> * Removing service and adding different cache interfaces along with event listener support Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle missingDoc issue Signed-off-by: Sagar Upadhyaya <[email protected]> * Changing listener logic, removing tiered cache integration with IRC Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding opensearch.internal tag for LoadAwareCacheLoader Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing thread safety issue Signed-off-by: Sagar Upadhyaya <[email protected]> * Remove compute function and event listener logic change for TieredCache Signed-off-by: Sagar Upadhyaya <[email protected]> * Making Cache.compute function private Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding javadoc and more test for cache.put Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding write locks to refresh API as well Signed-off-by: Sagar Upadhyaya <[email protected]> * Removing unwanted EventType class and refactoring one UT Signed-off-by: Sagar Upadhyaya <[email protected]> * Removing TieredCache interface Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]>

sgup432 added 2 commits June 4, 2024 14:31

Moving query recompute out of write lock

c06cdf2

Signed-off-by: Sagar Upadhyaya <[email protected]>

[Tiered Caching] Moving query recomputation logic outside of write lock

b0f6550

Signed-off-by: Sagar Upadhyaya <[email protected]>

sgup432 requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners June 11, 2024 16:51

Adding java doc for the completable map

1dcb7e9

Signed-off-by: Sagar Upadhyaya <[email protected]>

jainankitk reviewed Jun 17, 2024

View reviewed changes

modules/cache-common/src/main/java/org/opensearch/cache/common/tier/TieredSpilloverCache.java Show resolved Hide resolved

jainankitk reviewed Jun 17, 2024

View reviewed changes

Changes to call future handler only once per key

6cf2248

Signed-off-by: Sagar Upadhyaya <[email protected]>

minor UT refactor

3686d2c

Signed-off-by: Sagar Upadhyaya <[email protected]>

sohami approved these changes Jun 25, 2024

View reviewed changes

opensearch-ci-bot mentioned this pull request Jun 25, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteCloneIndexIT #14292

Open

sohami merged commit 563375d into opensearch-project:main Jun 25, 2024
31 checks passed

opensearch-trigger-bot bot added the backport-failed label Jun 25, 2024

sgup432 mentioned this pull request Jun 25, 2024

[BUG] org.opensearch.cache.common.tier.TieredSpilloverCacheTests.testComputeIfAbsentConcurrently is flaky #14544

Closed

sgup432 mentioned this pull request Jun 25, 2024

Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently #14550

Merged

2 tasks

opensearch-ci-bot mentioned this pull request Jun 26, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteSplitIndexIT #14296

Open

sgup432 mentioned this pull request Jun 26, 2024

[Backport 2.x] [Tiered Caching] Moving query recomputation logic outside of write lock #14563

Closed

1 task

sgup432 mentioned this pull request Jun 26, 2024

[Backport 2.x][Tiered Caching] Moving query recomputation logic outside of write lock #14565

Merged

1 task

finnegancarroll mentioned this pull request Jul 15, 2024

Make remote snapshot (local)block size configurable #14753

Closed

3 tasks

BrewTestBot mentioned this pull request Aug 8, 2024

opensearch 2.16.0 Homebrew/homebrew-core#180446

Merged

opensearch-ci-bot mentioned this pull request Sep 6, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreMultipartIT #15819

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tiered Caching] Moving query recomputation logic outside of write lock #14187

[Tiered Caching] Moving query recomputation logic outside of write lock #14187

sgup432 commented Jun 11, 2024

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 18, 2024

sgup432 commented Jun 25, 2024

github-actions bot commented Jun 25, 2024

sohami commented Jun 25, 2024

opensearch-trigger-bot bot commented Jun 25, 2024

[Tiered Caching] Moving query recomputation logic outside of write lock #14187

[Tiered Caching] Moving query recomputation logic outside of write lock #14187

Conversation

sgup432 commented Jun 11, 2024

Description

Related Issues

Check List

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 18, 2024

sgup432 commented Jun 25, 2024

github-actions bot commented Jun 25, 2024

sohami commented Jun 25, 2024

opensearch-trigger-bot bot commented Jun 25, 2024